Custom tests should also be transfomed, no? To cover the GPUArrays test suite.

Custom tests should also be transfomed, no? To cover the GPUArrays test suite.

I was debating that. In OpenCL.jl I currently have:

for name in keys(GPUArraysTestSuite.tests)
    test = "gpuarrays/$name"
    custom_tests[test] = test_transform(test, :(GPUArraysTestSuite.tests[$name](CLArray)))
end

Because I was envisioning folks wanting to have control over what their custom tests are doing.
But I can also see the argument to just want to apply the transform consistently (and you could make transform decisions based on the test name).

What about a tests::Dict{String, Expr} = ParallelTestRunner.discover_tests(pwd()) kwarg? That makes it clear that we default to running tests from files, makes it possible to override things by bringing your own tests (subsuming custom_tests), while obviating the need for a transform function as proposed here.

Would be a breaking change. Although I guess we could still support the old support_tests behavior in the mean time.

I guess that could then also do away with test_filter...

Closed in favor of #57